Systems and methods for highlighting content within media assets

ABSTRACT

Systems and methods are described herein for highlighting objects with a primary content that are likely to be of interest to a user viewing the primary content. More particularly, when the system receives a segment of primary content to be displayed on a user equipment device for consumption, the system analyzes the received segment to identify an object within the received segment. The system then checks a database storing supplemental content to determine whether supplemental content associated with the identified object is available. When supplemental content associated with the identified object is available within the database, the system modifies the received segment of the primary content to highlight the identified object and displays the modified segment of the primary content on the user equipment device for consumption.

BACKGROUND

This disclosure is directed to highlighting objects with a primarycontent that are likely to be of interest to a user viewing the primarycontent. More specifically, techniques are disclosed for modifyingdisplay of segments of a primary content to highlight objects within theprimary content to indicate availability of supplemental content (e.g.,advertising content, purchasing information, etc.) related to thehighlighted object.

SUMMARY

Businesses often advertise goods and services to consumers usingadvertisements presented to consumers via scheduled advertisement-breakswhile consumers are watching media assets such as a movie or aserialized program. In-video stream advertising such as the presentationof short video advertisements before displaying the primary content(e.g., a movie), during, and after the end of the primary content havebeen used by many streaming platforms. However, such advertisements maynot be of interest to the consumers and therefore lessen theireffectiveness. In one approach, modern media distribution systems oftenpresent personalized advertisement content identified based on a profileof the consumers (e.g., based on age, geographic location, etc.) viewingthese advertisements to in order to engage the consumers and increasethe effectiveness of the advertising content. However, consumers viewingmedia assets such as a movie or a serialized program often use scheduledadvertisement-breaks as opportunity to step away from the programming(e.g., to get snacks or to run other errands). Therefore, advertisementcontent which may be of interest to the consumers go unwatched,therefore reducing their effectiveness. Additionally, there may beproducts featured within the primary program being watched by theconsumers that they may be interested in. For example, a characterwearing a particular article of clothing (e.g., a jacket or a pair ofshoes) or driving a vehicle may pique the interest of the consumers.However, the consumers have to manually search for advertisement contentthat match the products featured within the primary program, whichwastes time, computing resources, and bandwidth due to a lack ofavailable information available for the primary programs. For example,requests for needless advertising content as the consumers are searchingfor other products unnecessarily consume limited available bandwidth.

To overcome these problems, systems and methods are provided herein forhighlighting objects with a primary content that are likely to be ofinterest to a user viewing the primary content. More particularly, whenthe system receives a segment of primary content to be displayed on auser equipment device for consumption, the system analyzes the receivedsegment to identify an object within the received segment. For example,the system may employ an object recognition algorithm to identify one ormore objects (e.g., an article of clothing, etc.) present within thereceived segment. The system then checks a database storing supplementalcontent (e.g., advertising content) to determine whether supplementalcontent associated with the identified object is available. Whensupplemental content associated with the identified object is availablewithin the database, the system modifies the display of the receivedsegment of the primary content to highlight the identified object. Insome aspects, the system may add a highlight in an overlay around aborder of the identified object. In another aspect, the identifiedobject can be highlighted by visually distinguishing the identifiedobject from other objects within the received segment (e.g., by addingan icon adjacent to the identified object, by providing a shaded overlayover the identified object, etc.). Finally, the system displays thesegment of the primary content with the highlighted object on the userequipment device for consumption. Accordingly, methods and systemsdisclosed herein allow businesses to present supplemental content (e.g.,advertising content) to users while they are engaged with the primarycontent. Moreover, users viewing the primary content can easily viewsupplemental content associated with objects within the segment of theprimary content that the users may be interested in.

In some aspects, the system can identify a plurality of objects withinthe received segment of the primary content. In such an aspect,highlighting every identified object within the received segment of theprimary content would overwhelm the user viewing the primary content anddistract from enjoying the primary content. Accordingly, the system mayprioritize one or two of the identified objects to be highlighted basedon user preferences. For example, the system may retrieve a web browsinghistory associated with the user consuming the primary content toidentify products that the user may wish to purchase. Upon determining amatch between an identified object from the received segment of theprimary content and a product identified from the retrieved web browsinghistory associated with the user, the system prioritizes the matchingidentified object. The system then modifies the received segment of theprimary content to highlight the highest priority objects and displaysthe modified segment on the user equipment device.

In some embodiments, modifying the received segment of the primarycontent includes providing a user selectable link. For example, the usercan use an input device such as a remote-control device to select ahighlighted object within the displayed segment of the primary content.In another embodiment, the user can provide a voice command using, forexample, a microphone, to select a highlighted object within thedisplayed segment of the primary content. The system, in response todetecting that the user has selected one of the highlight objects withinthe displayed segment of the primary content (for instance, by receivingdata indicative of a selection of the user selectable link), displaysthe supplemental content associated with the identified object selectedby the user.

In some embodiments, the system, in response to detecting that the userhas selected one of the highlight objects within the displayed segmentof the primary content, prevents an audio component associated with theprimary content from being output. For example, the system may mute theaudio of the primary content while simultaneously displaying thesupplemental content along with closed-captioning data associated withthe primary content. This ensures that the user can view thesupplemental content including the audio component of the supplementalcontent while simultaneously following the primary content. In anotherembodiment, the supplemental content is displayed during a segment ofthe primary content where there is no dialogue between characters. Forexample, the system may delay the presentation of supplemental contentuntil a segment of the primary content that does not include an audiocomponent is displayed. This allows the user to engage with thesupplemental content without missing the primary content.

In some aspects, the supplemental content is displayed on the userequipment device simultaneously with the primary content. For example,the primary content is displayed on a primary window of the userequipment device with the supplemental content being displayed on asecondary window as an overlay over the primary window. In another suchaspect, the playback of the primary content is paused while thesupplemental content is displayed on the primary window.

In an embodiments, the system, in response to detecting that the userhas selected one of the highlighted objects within the displayed segmentof the primary content, displays a user selectable option forinteracting with the supplemental content. In one aspect, the userselectable option is an option to interrupt playback of the primarycontent to display the supplemental content. For instance, playback ofthe primary content is paused while the supplemental content isdisplayed on the primary window of the user equipment device. In anotheraspect, the user selectable option is an option to simultaneouslydisplay the primary content and the supplemental content, wherein thesupplemental content is displayed in a picture-in-picture mode. In stillanother aspect, the user selectable option is an option to display thesupplemental content for display on a second user equipment device. Forexample, the system may transmit the supplemental content to a mobiledevice of the user so that the user can view the supplemental contentwithout interrupting the playback of the primary content.

In another aspect, the user selectable option in an option to view apurchase offer related to the identified object. For example, the systemmay present a Uniform Resource Locator (URL) to direct the user to a webresource where the user can purchase the identified object. In anotherembodiment, the system can display product reviews of the highlightedobject selected by the user. In an embodiment, the system displays oneor more of the user selectable options to be displayed on the userequipment device in response to detecting that the user has selected oneof the highlighted objects within the displayed segment of the primarycontent.

In accordance with some aspects of the subject matter disclosed herein,the primary content may have scheduled interruptions where advertisingcontent is conventionally displayed. In one aspect of the presentdisclosure, the system determines which of the presented user selectableoptions was selected by the user. The system then adjusts a frequency ofthe scheduled interruptions based on the determined selection of thepresented user selectable options. For example, when the user selectsthe option to pause the primary content and view the supplementalcontent on the primary window of the user equipment device, the systemremoves one or more of the scheduled interruptions.

In another aspect of the disclosure herein, the frequency of thescheduled interruptions is adjusted based on a duration of the userinteraction with the displayed supplemental content. For example, whenthe user merely interacts with the supplemental content for a minimalperiod of time (e.g., less than 5-10 seconds), the system may decide notto adjust the frequency of the scheduled interruption. In anotherembodiment, when the user interacts with the supplemental content for athreshold period of time (e.g., more than 30 seconds), the system mayremove one or more of the scheduled interruptions. In anotherembodiment, the system may reduce the amount of time scheduled for theinterruption to the primary content. Accordingly, the user isincentivized to interact with the supplemental content.

In accordance with one embodiment of the present disclosure, thesupplemental content displayed on the user equipment device ispersonalized based on the user viewing the supplemental content. Forinstance, the system can retrieve a purchase history of the user toidentify specific patterns favored by the user (e.g., specific colorsfor articles of clothing preferred by the user). The system thendisplays supplemental content such as an advertisement which featuresthe product in the specific color preferred by the user. In anotherembodiment, the system displays supplemental content which feature theidentified object (e.g., a specific pair of shoes worn by a character inthe primary content). In this way, businesses are more likely to engagethe consumers and increase effectiveness of their advertisements.Moreover, the users are more likely to identify products they areinterested in and easily view supplemental content related to thoseproducts.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the present disclosurewill be apparent upon consideration of the following detaileddescription, taken in conjunction with the accompanying drawings, inwhich like reference characters refer to like parts throughout, and inwhich:

FIG. 1 shows an illustrative system for modifying display of segments ofa primary content to highlight identified objects of interest, inaccordance with some embodiments of this disclosure;

FIG. 2 shows an illustrative user interface showing user selectableoptions for interacting with supplemental content associated with aselected highlighted object, in accordance with some embodiments of thisdisclosure;

FIG. 3 shows a block diagram of an illustrative user equipment deviceused to begin a live media stream, in accordance with some embodimentsof the disclosure;

FIG. 4 shows a block diagram of an illustrative media system, inaccordance with some embodiments of this disclosure;

FIG. 5 is a flowchart of a detailed illustrative process for modifyingdisplay of segments of a primary content to highlight identified objectsof interest, in accordance with some embodiments of this disclosure;

FIG. 6 shows a flowchart of a detailed illustrative process forproviding user selectable options for interacting with supplementalcontent, in accordance with some embodiments of this disclosure;

FIG. 7 is a flowchart of a detailed illustrative process 800 foradjusting a frequency of scheduled interruptions in the playback of theprimary content based on a type of interaction with the supplementalcontent, in accordance with some embodiments of this disclosure;

FIG. 8 is a flowchart of a detailed illustrative process 800 foradjusting a frequency of scheduled interruptions in the playback of theprimary content based on a duration of interaction with the supplementalcontent, in accordance with some embodiments of this disclosure; and

FIG. 9 is a flowchart of a detailed illustrative process for generatingpersonalized supplemental content, in accordance with some embodimentsof this disclosure.

DETAILED DESCRIPTION

FIG. 1 shows an exemplary system 100 for modifying display of segmentsof a primary content to highlight identified objects that are likely tobe of interest to a user viewing the primary content. As illustrated inFIG. 1 , control circuitry 102 receives segment 104 of primary contentto be displayed on user equipment device 108 for consumption. Forexample, control circuitry 102 receives a segment of the show“Billions”. In an embodiment, control circuitry 102, in response toreceiving a selection of a primary content (e.g., the show “Billions”),displays a user interface including a selectable option for providinginteractive supplemental content during playback of the selected primarycontent. For example, the user is offered a first option to requesthighlighted objects of interest during playback of the primary contentand a second option to consume the primary content without highlightingobjects of interest. In an embodiment, control circuitry 102 performsthe following processes in response to the user selecting the firstoption to request highlighting of objects of interest during playback ofthe primary content.

In an embodiment, control circuitry 102 analyzes the received segment toidentify an object within the received segment. For example, controlcircuitry 102 employs an object recognition algorithm (e.g., a knownobject recognition algorithm such as Microsoft Azure Computer Vision) toidentify one or more objects 112 present within the received segment. Inan embodiment, a voice recognition interface (such as the one describedbelow in connection with FIG. 3 ) can monitor utterances of the userduring playback of the primary content to identify objects that are ofinterest to the user. For example, the voice recognition interface candetermine that the user uttered the phrase, “that car looks nice!”. Inresponse, control circuitry 102 can use the object recognition algorithmdiscussed above to determine the presence of a vehicle within thereceived segment of the primary content. In the example illustrated inFIG. 1 , control circuitry 102 analyzes the received segment of the show“Billions” in the manner discussed above to identify objects such as avehicle 112-A (e.g., Mercedes sedan), a pair of shoes 112-B (e.g., NikeEbernon) worn by a character, and a tire 112-C of a second vehicle(e.g., Bridgestone).

Control circuitry 102 then checks database 106 storing supplementalcontent related to a plurality of objects to determine whethersupplemental content associated with the identified object is available.For example, control circuitry 102 checks database 106 to determinewhether supplemental content related to the identified vehicle, tires,or pair of shoes are available. When supplemental content associatedwith the identified object is available within database 106, controlcircuitry 102 modifies the received segment 104 of the primary content.Specifically, content circuitry 102 modifies the received segment 104 ofthe primary content to highlight the identified objects 112. Asillustrated in FIG. 1 , in one example, control circuitry 102 adds ahighlight around a border of the identified objects 112 in the modifiedsegment 110. In various other embodiments, control circuitry 102 canhighlight identified objects 112 by adding an icon next to theidentified objects within the modified segment of the primary content.In still other embodiments, control circuitry 102 can highlightidentified objects 112 in any suitable manner by visually distinguishingthe identified objects 112 from other objects within the receivedsegment (e.g., by shading an area of a frame where the identifiedobjects 112 are located).

Control circuitry 102 then displays the modified segment 110 of theprimary content on user equipment device 108 for consumption. Asillustrated in FIG. 1 , modified segment 114 displayed on usertelevision equipment 108 includes highlighted objects 112-A (a vehicle),112-B (a pair of shoes), and 112-C (tires of a vehicle).

Control circuitry 102 can identify a plurality of objects withinreceived segment 104 of the primary content. In such an aspect,highlighting every identified object within received segment 104 of theprimary content would overwhelm the user viewing the primary content anddistract from enjoying the primary content. Accordingly, controlcircuitry 102 prioritizes one or two of the identified objects to behighlighted based on user preferences. Specifically, control circuitry102 retrieves a web browsing history associated with the user consumingthe primary content to identify products that the user may wish topurchase. For example, control circuitry 102 can determine that the userhas recently searched extensively for new shoes and tires to purchase.In one embodiment, control circuitry 102 can retrieve social mediaactivity of the user viewing the primary content to identify otherinterests of the user. For example, control circuitry 102 may determine,based on the retrieved social media activity, that the user has recentlyliked pictures and advertisements related to Mercedes sedans.

Upon determining a match between an identified object from the receivedsegment of the primary content and the product identified from theretrieved user preferences, control circuitry 102 prioritizes thematching identified object. For example, as shown in FIG. 1 , theMercedes sedan 112-A, the pair of Nike Ebernon 112-B, and theBridgestone tires 112-C are prioritized. Similarly, objects within thereceived segment 104 that do not match the retrieved user preferencesare deprioritized. For example, objects such as a yacht shown in thebackground of the received segment 104 are deprioritized. Controlcircuitry 102 then modifies the received segment 104 of the primarycontent to highlight the highest priority objects (112-A, 112-B, and112-C) and displays modified segment 110 on user equipment device 108.

In some embodiments, modifying the received segment of the primarycontent includes providing a user selectable link. For example, the usercan use an input device such as a remote-control device to select ahighlighted object within the displayed modified segment 110 of theprimary content. In another embodiment, the user can provide a voicecommand using, for example, a microphone, to select a highlighted objectwithin the displayed segment of the primary content. In an embodiment, avoice recognition interface (such as the one described below inconnection with FIG. 3 ) can automatically monitor voice interactions ofthe user during playback of the primary content to detect an utterancecorresponding to an object.

As illustrated in FIG. 1 , the user selects the identified pair of shoes112-B from the displayed modified segment 110.

Control circuitry 102, in response to detecting that the user hasselected one of the highlight objects 112 within the displayed modifiedsegment 110 of the primary content (for instance, by receiving dataindicative of a selection of the user selectable link), displayssupplemental content 116 associated with the identified object 112selected by the user. As shown in FIG. 1 , control circuitry 102, inresponse to the user selecting the pair of shoes 112-B, displays anadvertisement (i.e., supplemental content 116) on user equipment device108.

In some embodiments, control circuitry 102, in response to detectingthat the user has selected one of the highlight objects 112 within thedisplayed segment 110 of the primary content, prevents an audiocomponent associated with the primary content from being output. Forexample, the system may mute the audio of the primary content whilesimultaneously displaying the supplemental content along withclosed-captioning data associated with the primary content. As shown inFIG. 1 , closed-captioning data 118 associated with the show “Billion”is displayed while simultaneously displaying the advertisement (i.e.,supplemental content 116) associated with the selected pair of shoes112-B. This ensures that the user can view the supplemental contentincluding the audio component of the supplemental content whilesimultaneously following the primary content. In another embodiment, thesupplemental content is displayed during a segment of the primarycontent where there is no dialogue between characters. For example, thesystem may delay the presentation of supplemental content until asegment of the primary content that does not include an audio componentis displayed. This allows the user to engage with the supplementalcontent without missing the primary content.

As illustrated in FIG. 1 , supplemental content 116 is displayed on userequipment device 108 simultaneously with the primary content. In theexample illustrated in FIG. 1 , the primary content is displayed onprimary window 120 of user equipment device 108 with supplementalcontent 116 being displayed on secondary window 122 as an overlay overprimary window 120. In another such aspect, the playback of the primarycontent is paused while supplemental content 116 is displayed on primarywindow 120.

In an embodiment, control circuitry 102, in response to detecting thatthe user has selected one of the highlighted objects within thedisplayed segment of the primary content, displays a user selectableoption for interacting with the supplemental content. FIG. 2 depicts anillustrative user interface 200 showing user selectable options 220 forinteracting with supplemental content associated with a selectedhighlighted object, in accordance with some embodiments of thisdisclosure.

In one aspect shown in FIG. 2 , user selectable option 220 is an optionto interrupt playback of the primary content to display the supplementalcontent. For instance, playback of the show “Billions” (i.e., primarycontent) is paused while the advertisement (i.e., supplemental content)for the pair of shoes 112-B is displayed on primary window 120 of userequipment device 108. In an embodiment where segments of the primarycontent are being received from a broadcast source, control circuitry102 buffers the primary content while displaying the supplementalcontent.

In another aspect, user selectable option 220 is an option tosimultaneously display the primary content and the supplemental content,where the supplemental content is displayed in a picture-in-picture modeon a secondary window overlaid over a primary window. For example, asillustrated in FIG. 1 , the show “Billions” (i.e., primary content) isdisplayed on primary window 120 while the advertisement (i.e.,supplemental content) for the pair of shoes 112-B is displayed onsecondary window 122 of user equipment device 108. Although notillustrated in FIG. 2 , in an embodiment, control circuitry 102 maygenerate a zoomed-in image of the identified object in the secondarywindow. In still another embodiment, the secondary window displaystextual description associated with the identified object instead of orin addition to the zoomed-in image of the identified object.

In still another aspect, user selectable option 220 is an option todisplay the supplemental content for display on a second user equipmentdevice. For example, control circuitry 102 displays an advertisement(i.e., the supplemental content) for the pair of shoes 112-B to a mobiledevice of the user so that the user can view the advertisement withoutinterrupting the playback of the show “Billions” (i.e., the primarycontent). In another aspect, user selectable option 220 is an option toview a purchase offer related to the identified object. As illustratedin FIG. 2 , control circuitry 102 presents a Uniform Resource Locator(URL) to direct the user to a web resource where the user can purchasethe identified pair of shoes 112-B. In another embodiment, controlcircuitry 102 can display product reviews of the highlighted objectselected by the user. As illustrated in FIG. 2 , control circuitry 102displays one or more of user selectable options 220 to be displayed onuser equipment device 108 in response to detecting that the user hasselected one of the highlighted objects 112 within the displayed segment114 of the primary content. As further illustrated in FIG. 2 , controlcircuitry 102 allows a user to scroll through user selectable options220 and select option 222 to purchase the pair of shoes 112-B.

In accordance with some aspects of the subject matter disclosed herein,the primary content may have scheduled interruptions during whichadvertising content is conventionally displayed. In one aspect of thepresent disclosure, control circuitry 102 determines which of thepresented user selectable options 220 was selected by the user. Controlcircuitry 102 then adjusts a frequency of the scheduled interruptionsbased on the determined selection 222 of the presented user selectableoptions 220. For example, when the user selects the option to pause theprimary content and view the supplemental content on the primary windowof the user equipment device, control circuitry 102 removes one or moreof the scheduled interruptions.

In another aspect of the disclosure herein, the frequency of thescheduled interruptions is adjusted based on a duration of the userinteraction with the displayed supplemental content. For example, whenthe user merely interacts with the supplemental content for a minimalperiod of time (e.g., less than 5-10 seconds), control circuitry 102determines not to adjust the frequency of the scheduled interruption. Inanother embodiment, when the user interacts with the supplementalcontent for a threshold period of time (e.g., more than 30 seconds),control circuitry 102 removes one or more of the scheduledinterruptions. In another embodiment, control circuitry 102 reduces theamount of time scheduled for the interruption to the primary content.Accordingly, the user is incentivized to interact with the supplementalcontent.

In one embodiment, the supplemental content displayed on user equipmentdevice 108 is personalized based on the user viewing the supplementalcontent. For instance, control circuitry 102 retrieves a purchasehistory of the user to identify specific patterns favored by the user(e.g., specific colors for articles of clothing preferred by the user).Control circuitry 102 then displays supplemental content such as anadvertisement which features the product in the specific color preferredby the user. In another embodiment, control circuitry 102 displayssupplemental content which feature the identified object (e.g., aspecific pair of shoes worn by a character in the primary content). Inthis way, businesses are more likely to engage the consumers andincrease effectiveness of their advertisements. Moreover, the users aremore likely to identify products they are interested in and easily viewsupplemental content related to those products.

In accordance with an embodiment, control circuitry 102 may be part ofuser television equipment 108. In such an embodiment, a processor(control circuitry 102) at the user television equipment 108 receivessegments of the primary content from a server along with a manifest fileassociated with the primary content. The control circuitry 102 at theuser television equipment 108 parses the manifest file to determinescheduled advertisement breaks. Moreover, control circuitry 102 at theuser television equipment 108 also retrieves supplemental content (e.g.,supplemental content 116 discussed above) from a remote database (e.g.,database 106 discussed above) and stitches together the primary contentalong with the supplemental content. Control circuitry 102 at the usertelevision equipment then modifies the display based on user selectionof one or more of the highlighted objects and/or user-selectableoptions. Although the above embodiment described control circuitry 102stitching together the primary content along with the supplementalcontent, it is not necessary to do so. In an embodiment, thesupplemental content is displayed as an overlay over the primarycontent. Moreover, as described below, the manifest file received atuser television equipment 108 may, in one embodiment, not include thesupplemental content (e.g., advertising content). Instead, a pluralityof URLs are received from an advertising server corresponding to theidentified objects separate from the manifest file as discussed below.

In another embodiment, control circuitry 102 may be part of a mediacontent server remote from user television equipment 108. In such anembodiment, the media content server receives segments of the primarycontent from a media content owner server (e.g., Showtime serverproviding segments of the program “Billions”). The media content serveralso receives advertising content from an advertisement server to bedisplayed during the display of the primary content. The media contentserver then stitches together the advertising content with the primarycontent and generates a manifest file for the combined content. Thecombined content along with the manifest file is then transmitted to,for instance, the user television equipment 108. Additional detail aboutthe structure and operation of control circuitry 102 will be describedin greater detail below in connection with the discussion of FIGS. 3 and4 .

In another embodiment, media content server does not stitch together theadvertising content with the primary content. Instead, media contentserver generates an Extensible Markup Language (XML) file or a file thatstores simple data structures and objects in JavaScript Object Notation(JSON) format. More particularly, media content server generates an XMLfile or JSON file which includes URLs corresponding to the supplementalcontent (e.g., advertising content) corresponding to various objectswithin the received segment of the primary content. In addition, mediacontent server can include, within the XML file or JSON file,information indicative of location of an object of interest within eachframe of the segment of the primary content (e.g., pixel locationinformation where the identified object is to be displayed). Moreover,media content server can further include, within the XML file or JSONfile, information indicative of location where and how the highlight isto be generated for display. For example, media content server canprovide information instructing user television equipment 108 togenerate for display a yellow colored rectangular box at a locationcorresponding to an object of interest within the segment of the primarycontent. In such an embodiment, user television equipment 108, inresponse to receiving the manifest file and the XML or JSON file, canmerely generate for display the primary content and the highlightingbased on display signals received from the media content server. Inresponse to receiving a user selection of a highlighted object, usertelevision equipment can retrieve supplemental content using URLinformation stored in the received XML or JSON file. In anotherembodiment, in response to receiving a user selection of a highlightedobject, user television equipment can transmit a request to the mediacontent server for supplemental content. In such an embodiment, mediacontent server can provide the supplemental content to user televisionequipment 108 along with information indicating where and how thesupplemental content is to be displayed in an XML or JSON fileaccompanying the supplemental content data.

FIG. 3 depicts a generalized embodiment of an illustrative device (e.g.,user equipment device 204 or user equipment device 220) that is used tomodify the received segments of the primary content. User equipmentdevice 300 may be any of a plurality of user devices such as asmartphone, a tablet, personal computer, etc. (discussed further belowwith respect to FIG. 4 ). User equipment device 300 may transmit orreceive the live media stream data via input/output (hereinafter “I/O”)path 302. I/O path 302 may provide the live media stream data (e.g.,content item available over LAN or WAN, and the like) and data tocontrol circuitry 304, which includes processing circuitry 306 andstorage 308. Control circuitry 304 may be used to send and receivecommands, requests, and other suitable data using I/O path 302. I/O path302 may connect control circuitry 304 (and specifically processingcircuitry 306) to one or more communications paths (described below inrelation to FIG. 4 ). I/O functions may be provided by one or more ofthese communications paths but are shown as a single path in FIG. 3 toavoid overcomplicating the drawing.

Control circuitry 304 may be based on any suitable processing circuitrysuch as processing circuitry 306. Processing circuitry should beunderstood to mean circuitry based on one or more microprocessors,microcontrollers, digital signal processors, programmable logic devices,field-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), etc., and may include a multi-core processor (e.g.,quad-core). In some embodiments, processing circuitry may be distributedacross multiple separate processor or processing units, for example,multiple of the same type of processing units (e.g., two Intel Core i7processors) or multiple different processors (e.g., Ryzen processor withintegrated CPU and GPU processing cores) or may include a multi-coreprocessor (e.g., dual-core, quad-core, hexa-core, or any suitable numberof cores) or supercomputer. In some embodiments, control circuitry 304executes instructions for an application stored in memory (e.g., memory308). Specifically, control circuitry 304 may be instructed by a mediaapplication to perform the functions discussed above and below. Forexample, the media application may provide instructions to controlcircuitry 304 to modify segments of the primary content to highlightcertain objects. Moreover, the media application may also collect userpreference information and select suitable supplemental content. In someimplementations, any action performed by control circuitry 304 may bebased on instructions received from the media application.

Control circuitry 304 may include tuning circuitry, such as one or moreanalog tuners, one or more MP3 decoders or other digital decodingcircuitry, or any other suitable tuning or audio circuits orcombinations of such circuits. Encoding circuitry (e.g., for convertinganalog or digital signals to signals for storage in memory 308) may alsobe provided. Control circuitry 304 may also include scaler circuitry forupconverting and down converting content item into the preferred outputformat of user equipment device 300, and converter circuitry forconverting between digital and analog signals. The tuning and encodingcircuitry may be used by user equipment device 300 to receive, play, andbuffer content item. The circuitry described herein, including forexample, the tuning, audio generating, encoding, decoding, encrypting,decrypting, scaler, and analog/digital circuitry, may be implementedusing software running on one or more general purpose or specializedprocessors. If storage 308 is provided as a separate device from userequipment device 300, the tuning and encoding circuitry may beassociated with storage 308.

Storage 308 may be any device for storing electronic data, such asrandom-access memory, solid state devices, quantum storage devices, harddisk drives, non-volatile memory or any other suitable fixed orremovable storage devices, and/or any combination of the same. Controlcircuitry 304 may allocate portions of storage 308 for various purposessuch as caching application instructions, recording media assets,storing portions of a media asset, buffering segments of media, etc. Asdescribed herein, storage 308 may be used to store one or more LUTsstoring a number of MAC addresses associated with a plurality of userequipment devices and their corresponding profile information.

A user may send instructions to control circuitry 304 using user inputinterface 310. User input interface 310 may be any suitable user inputinterface, such as a touchscreen as shown in FIG. 1 , mouse, trackball,keypad, keyboard, touchpad, stylus input, joystick, voice recognitioninterface, or other user input interfaces. Instructions to controlcircuitry 304 may be transmitted through I/O path 302, that couldconsist of a video tracking and detection mechanism, Internet of Things(IoT) and home automation triggers, emergency alert systems, andsoftware or hardware communication pipelines and/or notificationcenters.

Display 312 may be provided as a stand-alone device or integrated withother elements of each one of user equipment device 300. For example,display 312 may be a touchscreen or touch-sensitive display, aprojector, or a casting device. In such circumstances, user inputinterface 310 may be integrated with or combined with display 312.Display 312 may be one or more of a monitor, a television, aliquid-crystal display (LCD) for a mobile device, silicon display, e-inkdisplay, light-emitting diode (LED) display, or any other suitableequipment for displaying visual images. Graphics processing circuitrymay generate the output to the display 312. In some embodiments, thegraphics processing circuitry may be external to processing circuitry306 (e.g., as a graphics processing card that communicates withprocessing circuitry 306 via I/O path 302) or may be internal toprocessing circuitry 306 or control circuitry 304 (e.g., on a samesilicone die as control circuitry 304 or processing circuitry 306). Insome embodiments, the graphics processing circuitry may be used toreceive, display, and play the media asset.

Speakers 314 may be provided as integrated with other elements of userequipment device 300 or may be stand-alone units. The audio component ofvideos and other media asset displayed on display 312 may be playedthrough speakers 314. In some embodiments, the audio may be distributedto a receiver (not shown), which processes and outputs the audio viaspeakers 314. The speakers 314 may be part of, but not limited to, ahome automation system. In some embodiments, speakers 314 may alsoinclude a microphone to receive audio input from the first user startingthe live media stream.

The media application may be implemented using any suitablearchitecture. For example, it may be a stand-alone application whollyimplemented on user equipment device 300. The user interface applicationand/or any instructions for performing any of the embodiments discussedherein may be encoded on computer-readable media. Computer-readablemedia includes any media capable of storing data.

FIG. 4 depicts an exemplary media system in accordance with someembodiments of the disclosure in which user equipment device 108 anduser equipment device 300 can be implemented in system 400 of FIG. 4 asuser television equipment 402, user computer equipment 404, wirelessuser communications device 406, or any other type of user equipmentsuitable for accessing media. For simplicity, these devices may bereferred to herein collectively as user equipment. User equipment, onwhich the media application is implemented, may function as astand-alone device or may be part of a network of devices. Variousnetwork configurations of devices may be implemented and are discussedin more detail below.

User television equipment 402 may include circuitry for receivingcontent over the Internet, a television set, a digital storage device,or other user television equipment. One or more of these devices may beintegrated to be a single device, if desired. User computer equipment404 may include a PC, a laptop, a streaming content item aggregator, aPC media center, or other user computer equipment. It may includedevices like digital assistance, smart speakers, and/or home automation.Wireless user communications device 406 may include a smartphone, aportable video player, a portable music player, a portable gamingmachine, a tablet, a wireless streaming device or other wireless device.It should be noted that the lines are blurred when trying to classify adevice as one of the above devices and one device may be categorizedinto one or more of the categories listed above.

In system 400, there is typically more than one of each type of userequipment but only one of each is shown in FIG. 4 to avoidovercomplicating the drawing. In addition, each user may utilize morethan one type of user equipment (e.g., a user may have a computer and atablet) and also more than one of each type of user equipment device(e.g., a user may have multiple television sets).

The user equipment may be coupled to communications network 414. Namely,user television equipment 402, user computer equipment 404, and wirelessuser communications device 406 are coupled to communications network 414via communications paths 408, 410, and 412, respectively. Communicationsnetwork 414 is used by the user equipment to transmit or receive thelive media stream. Communications network 414 may be one or morenetworks including the Internet, a mobile phone network, ad-hoc network,a Local Area network (LAN), or other types of communications network orcombination of communications networks. Paths 408, 410, and 412 mayseparately or together include one or more communications paths,including any suitable wireless communications path. Paths 408 and 410are drawn as solid lines to indicate they are wireless paths and path412 is drawn as dotted line to indicate it is a wired path.Communications with the user equipment may be provided by one or more ofthese communications paths but are shown as a single path in FIG. 4 toavoid overcomplicating the drawing. The user equipment devices maycommunicate with each other directly through an indirect path viacommunications network 414.

System 400 includes content item source 416 coupled to communicationsnetwork 414 via communications path 420. Path 420 may include any of thecommunications paths described above in connection with paths 408, 410,and 412. Communications with the content item source 416 may beexchanged over one or more communications paths but are shown as asingle path in FIG. 4 to avoid overcomplicating the drawing. Inaddition, there may be more than one of content item source 416, butonly one is shown in FIG. 4 to avoid overcomplicating the drawing.Although communications between source 416 with user equipment 402, 404,and 406 are shown as through communications network 414, in someembodiments, source 416 may communicate directly with user equipmentdevices 402, 404, and 406 via communications paths (not shown) such asthose described above in connection with paths 408, 410, and 412.

Content item source 416 may include one or more types of mediadistribution equipment such as a media server, cable system headend,satellite distribution facility, intermediate distribution facilitiesand/or servers, Internet providers, on-demand media servers, and othermedia providers. Content item source 416 may be the originator of mediacontent item or may not be the originator of media content item. Contentitem source 416 may also include a remote media server used to storedifferent types of media content item (including live media stream data(e.g., a plurality of frames) uploaded by a user), in a location remotefrom any of the user equipment.

System 400 is intended to illustrate a number of approaches, or networkconfigurations, by which user equipment devices and sources of mediacontent item and guidance data may communicate with each other for thepurpose of accessing media and data related to the media. Theconfiguration of the devices and paths in system 400 may change withoutdeparting from the scope of the present disclosure.

FIG. 5 depicts a flowchart of a detailed illustrative process 500 formodifying display of segments of a primary content to highlightidentified objects of interest. In various embodiments, the individualsteps of process 500 may be implemented by one or more components of thedevices and systems of FIGS. 1-4 . Although the present disclosure maydescribe certain steps of process 500 (and of other processes describedherein) as being implemented by certain components of the devices andsystems of FIGS. 1-4 , this is for purposes of illustration only, and itshould be understood that other components of the devices and systems ofFIGS. 1-4 may implement those steps instead. For example, the steps ofprocess 500 may be executed at control circuitry 102 of FIG. 1 .

At 502, control circuitry 102 receives a segment of primary content(e.g., segment 104) to be displayed on a user equipment device (e.g.,television 108) for consumption. For example, control circuitry 102receives a segment of the show “Billions”. At 504, control circuitry 102analyzes the received segment to identify an object within the receivedsegment. In one embodiment, control circuitry 102 employs an objectrecognition algorithm to identify one or more objects 112 present withinthe received segment 104. In the example illustrated in FIG. 1 , controlcircuitry 102 analyzes the received segment of the show “Billions” toidentify objects such as a vehicle 112-A (e.g., Mercedes sedan), a pairof shoes 112-B (e.g., Nike Ebernon) worn by a character, and a tire112-C of a second vehicle (e.g., Bridgestone). In additionalembodiments, control circuitry 102 can analyze, for example, a colorgradient of each frame of the received segment 104 to identify regionsof the frame with richer detail in order to identify objects presentwithin the received segment 104.

At 506, control circuitry 102 then checks a database (e.g., database106) storing supplemental content related to a plurality of objects todetermine whether supplemental content associated with the identifiedobject is available. For example, control circuitry 102 checks database106 to determine whether supplemental content related to the identifiedvehicle, tires, or pair of shoes are available. When supplementalcontent associated with the identified object is available withindatabase 106 (YES at 506), process 500 proceeds to 510 where controlcircuitry 102 modifies the received segment 104 of the primary content.Specifically, content circuitry 102 modifies the received segment 104 ofthe primary content to highlight the identified objects 112. Asillustrated in FIG. 1 , in one example, control circuitry 102 adds ahighlight around a border of the identified objects 112 in the modifiedsegment 110. In various other embodiments, control circuitry 102 canhighlight identified objects 112 by adding an icon next to theidentified objects within the modified segment of the primary content.

At 512, control circuitry 102 then displays the modified segment 110 ofthe primary content on user equipment device (e.g., television 108) forconsumption. As illustrated in FIG. 1 , modified segment 114 displayedon the user equipment device (e.g., television 108) includes highlightedobjects 112-A (a vehicle), 112-B (a pair of shoes), and 112-C (tires ofa vehicle).

If, on the other hand, control circuitry 102 determines thatsupplemental content associated with the identified object is notavailable (NO at 506), process 500 proceeds to 508 where controlcircuitry 102 displays the received segment of the primary contentwithout any modification. In an embodiment, control circuitry 102, inresponse to determining that supplemental content associated with theidentified object is not available (NO at 506), nevertheless modifiesthe received segment to highlight the identified object. In such anembodiment, upon detecting that the user has selected a highlightedobject (described in greater detail below in connection with thediscussion of FIG. 6 ) that does not have corresponding supplementalcontent available, generates a search string which includes theidentified object and displays the search results for the generatedsearch string on the user equipment device. In an embodiment, the searchresults can be displayed on a second user equipment device (e.g., amobile phone) so as to avoid interrupting playback of the primarycontent.

FIG. 6 is a flowchart of a detailed illustrative process 600 forpresenting supplemental content to the user, in accordance with someembodiments of this disclosure. In various embodiments, the individualsteps of process 600 may be implemented by one or more components of thedevices and systems of FIGS. 1-4 . Although the present disclosure maydescribe certain steps of process 600 (and of other processes describedherein) as being implemented by certain components of the devices andsystems of FIGS. 1-4 , this is for purposes of illustration only, and itshould be understood that other components of the devices and systems ofFIGS. 1-4 may implement those steps instead. For example, the steps ofprocess 600 may be executed at control circuitry 102 of FIG. 1 .

At 602, control circuitry 102 displays the modified segment of theprimary content which includes highlighted objects, where thehighlighted objects each include a user selectable link. For example,the user can use an input device such as a remote-control device toselect a highlighted object within the displayed modified segment 110 ofthe primary content. In another embodiment, the user can provide a voicecommand using, for example, a microphone, to select a highlighted objectwithin the displayed segment of the primary content.

At 604, control circuitry 102 determines whether the user has selectedone of the highlighted objects. For example, as discussed above, theuser can use a remote-control device or a voice recognition interface toselect one of the highlighted objects. If control circuitry 102determines that the user has not selected one of the highlighted objectsincluded in the modified segment (NO at 604), process 600 proceeds to606 where the control circuitry 102 begins analyzing a next segment ofthe primary content.

On the other hand, process 600, in response to detecting that the userhas selected one of the highlight objects 112 within the displayedmodified segment 110 of the primary content (for instance, by receivingdata indicative of a selection of the user selectable link) (YES at604), process 600 proceeds to 608 where control circuitry 102 determineswhich of the highlighted objects selected by the user. For example,control circuitry 102 receives data indicating which of the highlightedobjects was selected by the user from the voice recognition interface.At 610, control circuitry 102 displays a plurality of user selectableoptions for consuming supplemental content associated with the selectedobject. As discussed above in connection with FIG. 2 , the plurality ofuser selectable options can include (i) a first user selectable optionto interrupt playback of the primary content to display the supplementalcontent; (ii) a second user selectable to simultaneously display theprimary content and the supplemental content (e.g., the supplementalcontent is displayed in a picture-in-picture mode or an overlay mode);(iii) a third user selectable option to display the supplemental contentfor display on a second user equipment device; or (iv) a fourth userselectable option to view a purchase offer related to the selectedobject. Based on a user selection at 610 from one of the displayed userselectable options, process 600 proceeds to one of 612, 614, 616, or 618corresponding to the first, second, third, or fourth user selectableoption, respectively. In an embodiment, the user can select more thanone option for consuming the supplemental content. For example, the usercan select the options to view the supplemental content in an overlaymode on television 108, while simultaneously receiving a link topurchase the selected object on a second user equipment device.

FIG. 7 is a flowchart of a detailed illustrative process 700 foradjusting a frequency of scheduled interruptions in the playback of theprimary content, in accordance with some embodiments of this disclosure.In various embodiments, the individual steps of process 700 may beimplemented by one or more components of the devices and systems ofFIGS. 1-4 . Although the present disclosure may describe certain stepsof process 700 (and of other processes described herein) as beingimplemented by certain components of the devices and systems of FIGS.1-4 , this is for purposes of illustration only, and it should beunderstood that other components of the devices and systems of FIGS. 1-4may implement those steps instead. For example, the steps of process 700may be executed at control circuitry 102 of FIG. 1 .

At 702, control circuitry 102 receives a selection of a primary contentfor display. For example, as illustrated in FIG. 1 , a user selects theprogram “Billions” to view using the television equipment 108. At 704,control circuitry 102 determines whether there are pre-scheduledadvertisement breaks included in the display of the primary content. Forexample, information about scheduled in-video stream advertising such asthe presentation of short video advertisements during display of theprimary content (e.g., the program “Billions”) can be retrieved from amanifest file (e.g., a M3U8 Audio playlist file) for the primarycontent. If, at 704, control circuitry 102 determines that there are noscheduled advertisement breaks within the primary content (NO at 704),process 700 proceeds to 706. At 706, control circuitry 102 modifies asegment of the primary content to highlight an identified object, asdiscussed above in greater detail. At 708, control circuitry 102displays the modified segment of the primary content on the userequipment device.

On the other hand, when control circuitry 102 determines that there arepre-scheduled advertisement breaks included in the display of theprimary content (YES at 704), process 700 proceeds to 710 where controlcircuitry 102 modifies a segment of the primary content to highlight anidentified object, as discussed above in greater detail. At 712, controlcircuitry 102 displays the modified segment of the primary content onthe user equipment device. At 714, control circuitry 102 identifieswhich highlighted object has been selected by the user. For example,control circuitry 102 receives data indicating which of the highlightedobjects was selected by the user from the voice recognition interface.

At 716, control circuitry 102 displays a plurality of user selectableoptions for consuming supplemental content associated with the selectedobject. As discussed above in connection with FIG. 2 , the plurality ofuser selectable options can include (i) a first user selectable optionto interrupt playback of the primary content to display the supplementalcontent; (ii) a second user selectable to simultaneously display theprimary content and the supplemental content (e.g., the supplementalcontent is displayed in a picture-in-picture mode or an overlay mode);(iii) a third user selectable option to display the supplemental contentfor display on a second user equipment device; or (iv) a fourth userselectable option to view a purchase offer related to the selectedobject.

At 718, control circuitry 102 determines whether the user selected thefirst user selectable option to interrupt playback of the primarycontent to display the supplemental content (i.e., the first userselectable option). If, at 718, control circuitry 102 determines thatthe user selected the first user selectable option to interrupt playbackof the primary content to display the supplemental content (YES at 718),process 700 proceeds to 720 where control circuitry 102 removes at leastone of the scheduled advertisement breaks from the primary content. Forexample, control circuitry 102 modifies the manifest file for theprimary content to remove an advertisement break. If, on the other hand,control circuitry 102 determines that the user has not selected thefirst user selectable option to interrupt playback of the primarycontent to display the supplemental content (NO at 718), process 700proceeds to 722 where control circuitry 102 maintains the schedule ofadvertisement breaks within the primary content.

In other embodiments, the scheduled advertisement breaks may be modifiedin a plurality of ways. For example, often there are multipleadvertisements scheduled to be displayed during a scheduledadvertisement break. In such an instance, based on the selection of theuser selectable option, control circuitry modifies the number ofadvertisements displayed during an advertisement breaks while stillmaintaining the schedule of advertisement breaks. In an embodiment, theduration of each of the advertisement break is modified based on theuser selection of one or more of the user selectable options forinteracting with the supplemental content.

FIG. 8 is a flowchart of a detailed illustrative process 800 foradjusting a frequency of scheduled interruptions in the playback of theprimary content, in accordance with some embodiments of this disclosure.In various embodiments, the individual steps of process 800 may beimplemented by one or more components of the devices and systems ofFIGS. 1-4 . Although the present disclosure may describe certain stepsof process 800 (and of other processes described herein) as beingimplemented by certain components of the devices and systems of FIGS.1-4 , this is for purposes of illustration only, and it should beunderstood that other components of the devices and systems of FIGS. 1-4may implement those steps instead. For example, the steps of process 800may be executed at control circuitry 102 of FIG. 1 .

At 802, control circuitry 102 displays the modified segment of theprimary content which includes highlighted objects, where thehighlighted objects each include a user selectable link. For example,the user can use an input device such as a remote-control device toselect a highlighted object within the displayed modified segment 110 ofthe primary content. In another embodiment, the user can provide a voicecommand using, for example, a microphone, to select a highlighted objectwithin the displayed segment of the primary content.

At 804, control circuitry 102 determines whether the user has selectedone of the highlighted objects. For example, as discussed above, theuser can use a remote-control device or a voice recognition interface toselect one of the highlighted objects. If control circuitry 102determines that the user has not selected one of the highlighted objectsincluded in the modified segment (NO at 804), process 800 proceeds to806 where the control circuitry 102 begins analyzing a next segment ofthe primary content.

On the other hand, process 800, in response to detecting that the userhas selected one of the highlighted objects 112 within the displayedmodified segment 110 of the primary content (for instance, by receivingdata indicative of a selection of the user selectable link) (YES at804), process 800 proceeds to 808. At 808, control circuitry 102calculates a duration of the interaction with the supplemental contentassociated with the selected highlighted object. For example, controlcircuitry 102 determines whether the user viewed an entire advertisementfor the selected highlighted object or merely skipped the advertisementafter viewing the first few seconds of the advertisement.

At 810, control circuitry 102 determines whether the duration ofinteraction was greater than a threshold period of time. In anembodiment, the threshold period of time may be, for example, 30seconds. If, at 810, control circuitry 102 determines that the userinteracted with the supplemental content for a duration greater than thethreshold period of time (YES at 810), process 800 proceeds to 814 wherecontrol circuitry 102 removes at least one of the scheduledadvertisement breaks from the primary content. For example, controlcircuitry 102 modifies the manifest file for the primary content toremove an advertisement break. If, on the other hand, control circuitry102 determines that the user has not interacted with the supplementalcontent for a duration less than or equal to the threshold period oftime (NO at 810), process 800 proceeds to 812 where control circuitry102 maintains the schedule of advertisement breaks within the primarycontent.

In other embodiments, the threshold period of time can be dynamicallyset based on the selection of one of the plurality of user selectableoptions presented to the user (as discussed above in connection withFIG. 2 ). For example, the frequency of the scheduled advertisementbreaks is adjusted based on a duration of the user interaction with thedisplayed supplemental content as well as the type of user interactionwith the displayed supplemental content. For example, when the usermerely interacts with the supplemental content for a minimal period oftime (e.g., less than 30 seconds) when the supplemental content is anadvertisement, control circuitry 102 determines not to adjust thefrequency of the scheduled interruption. In another embodiment, when theuser merely interacts with the supplemental content for a minimal periodof time (e.g., less than 30 seconds) but the supplemental content is alink to purchase the identified object and the user completes thepurchase, control circuitry 102 removes one or more of the scheduledinterruptions.

FIG. 9 is a flowchart of a detailed illustrative process 900 forgenerating personalized supplemental content, in accordance with someembodiments of this disclosure. In various embodiments, the individualsteps of process 900 may be implemented by one or more components of thedevices and systems of FIGS. 1-4 . Although the present disclosure maydescribe certain steps of process 900 (and of other processes describedherein) as being implemented by certain components of the devices andsystems of FIGS. 1-4 , this is for purposes of illustration only, and itshould be understood that other components of the devices and systems ofFIGS. 1-4 may implement those steps instead. For example, the steps ofprocess 900 may be executed at control circuitry 102 of FIG. 1 .

At 902, control circuitry 102 displays the modified segment of theprimary content which includes highlighted objects, where thehighlighted objects each include a user selectable link. For example,the user can use an input device such as a remote-control device toselect a highlighted object within the displayed modified segment 110 ofthe primary content. In another embodiment, the user can provide a voicecommand using, for example, a microphone, to select a highlighted objectwithin the displayed segment of the primary content.

At 904, control circuitry 102 determines whether the user has selectedone of the highlighted objects. For example, as discussed above, theuser can use a remote-control device or a voice recognition interface toselect one of the highlighted objects. If control circuitry 102determines that the user has not selected one of the highlighted objectsincluded in the modified segment (NO at 904), process 900 proceeds to906 where the control circuitry 102 begins analyzing a next segment ofthe primary content.

On the other hand, process 900, in response to detecting that the userhas selected one of the highlighted objects 112 within the displayedmodified segment 110 of the primary content (for instance, by receivingdata indicative of a selection of the user selectable link) (YES at904), process 900 proceeds to 908. At 908, control circuitry 102retrieves a web browsing history associated with the user consuming theprimary content to identify products that the user may wish to purchase.In addition, at 908, control circuitry 102 can retrieve social mediaactivity of the user viewing the primary content to identify otherinterests of the user. At 910, control circuitry 102 determines contentpreferences for the user based on the retrieved web browser activity andthe social media activity associated with the user. For example, controlcircuitry 102 can determine that the user has recently searchedextensively for new shoes and tires to purchase. Moreover, controlcircuitry 102 can determine, based on the retrieved social mediaactivity, that the user has recently liked pictures and advertisementsrelated to Mercedes sedans.

At 912, control circuitry 102 modifies the available supplementalcontent retrieved from database 106 based on the determined contentpreferences. For example, control circuitry 102, based on prior shoepurchases of the user, can infer that the user prefers sneakers in thecolor blue. Accordingly, at 912, control circuitry 102 modifies thesupplemental content (e.g., an advertisement for the selected pair ofshoes) to include sneakers in the color blue in order to match theuser's preferences. In another example, where the supplemental contentis an advertisement for tires, control circuitry 102 can modify thesupplemental content to feature the tires on a vehicle owned by the userin order to engage the user and increase the effectiveness of theadvertisement. In another embodiment, control circuitry 102 modifies thesupplemental content based on a geographic location of the user.

In an embodiment, control circuitry 102 can automatically add theselected highlighted object to a shopping wish list associated with theuser. In another embodiment, control circuitry 102 provides the user anoption to “like” or “dislike” one or more of the highlighted objects.Control circuitry 102, maintains a preference indication associated witha highlighted object (e.g., selection of a “like” or “dislike” option)in order to inform future analysis of received segments of the primarycontent being consumed by the user. For example, when a user “dislikes”an automobile, control circuitry 102 deprioritizes highlighting ofautomobiles in additional segments of the primary content.

In some instances, users may be interested in viewing scheduledadvertising content but nevertheless miss it because they are unaware ofthe specific advertisements scheduled to be displayed during an upcomingadvertisement break. Accordingly, in some embodiments, control circuitry102 highlights objects corresponding to an upcoming scheduledadvertisement in order to allow the user to view the desiredadvertisement before the scheduled advertisement break. In someembodiments, control circuitry 102 modifies the segment of the primarycontent to highlight objects corresponding to an upcoming scheduledadvertisement in order to inform the user of the upcoming scheduledadvertisement without providing the option to view the advertisementimmediately. For example, in an embodiment where the primary content isbeing received from a broadcast source, control circuitry 102 maydetermine to display the desired advertisement during the scheduledadvertising break instead of buffering the broadcast content whiledisplaying the desired advertisement immediately. In still anotherembodiment, highlighted objects having supplemental content availablefor immediate viewing may be visually distinguished from objectshighlighted to indicate an upcoming advertisement scheduled to bedisplayed during the next advertising break.

The processes described above are intended to be illustrative and notlimiting. One skilled in the art would appreciate that the steps of theprocesses discussed herein may be related causally (i.e., in response),omitted, modified, combined, and/or rearranged, and any additional stepsmay be performed without departing from the scope of the invention. Moregenerally, the above disclosure is meant to be exemplary and notlimiting. Only the claims that follow are meant to set bounds as to whatthe present invention includes. Furthermore, it should be noted that thefeatures and limitations described in any one embodiment may be appliedto any other embodiment herein, and flowcharts or examples relating toone embodiment may be combined with any other embodiment in a suitablemanner, done in different orders, or done in parallel. In addition, thesystems and methods described herein may be performed in real time. Itshould also be noted that the systems and/or methods described above maybe applied to, or used in accordance with, other systems and/or methods.

1. A method comprising: receiving a segment of primary content to bedisplayed on a user equipment device for consumption; analyzing thereceived segment to identify an object within the received segment ofthe primary content; determining whether supplemental content associatedwith the identified object is available; in response to determining thatsupplemental content associated with the identified object is available:modifying the received segment of the primary content to highlight theidentified object within the received segment of the primary content,and displaying the modified segment of the primary content on the userequipment device for consumption.
 2. The method of claim 1, whereinmodifying the received segment of the primary content includes providinga user selectable link; and the method further comprising: receivingdata indicative of a selection of the user selectable link; and inresponse to receiving data indicative of the selection of the userselectable link, displaying the supplemental content associated with theidentified object on the user equipment device for consumption.
 3. Themethod of claim 2, further comprising: in response to receiving dataindicative of the selection of the user selectable link: preventing anaudio component associated with the primary content from being output;and simultaneously displaying closed-captioning data associated with theprimary content and the supplemental content on the user equipmentdevice.
 4. The method of claim 1, wherein modifying the received segmentof the primary content to highlight the identified object within thereceived segment of the primary content comprises: determining whetherthe received segment of the primary content includes an audio component;and modifying the received segment of the primary content to highlightthe identified object in response to determining that the receivedsegment of the primary content does not include an audio component. 5.The method of claim 2, wherein the supplemental content is a video mediaasset; and the method further comprises: simultaneously displaying theprimary content in a primary window and the supplemental content on asecondary window as an overlay over the primary window.
 6. The method ofclaim 2, further comprising: in response to receiving data indicative ofthe selection of the user selectable link, presenting a plurality ofuser selectable options for consuming the supplemental content.
 7. Themethod of claim 6, wherein presenting the plurality of user selectableoptions includes one or more of: presenting a first option to interruptplayback of the primary content to display the supplemental content;presenting a second option to simultaneously display the primary contentand the supplemental content, wherein the supplemental content isdisplayed in a picture-in-picture mode; presenting a third option totransmit the supplemental content for display on a second user equipmentdevice; and presenting a fourth option to view a purchase offer relatedto the identified object.
 8. The method of claim 7, wherein consumptionof the primary content is scheduled to be interrupted to displaypromotional content; and the method further comprising: determiningwhich of the user selectable options was selected on the first equipmentdevice; and adjusting a frequency of the scheduled promotional contentduring consumption of the primary content based on selection of the userselectable options.
 9. The method of claim 7, wherein consumption of theprimary content is scheduled to be interrupted to display promotionalcontent; and the method further comprising: determining which of theuser selectable options was selected on the first equipment device; andadjusting a frequency of the scheduled promotional content duringconsumption of the primary content based on a duration of interactionwith the supplemental content.
 10. The method of claim 2, wherein thesupplemental content to be displayed on the user equipment device isdetermined based on a content consumption profile associated with afirst user consuming the primary content.
 11. A system comprising: astorage database storing supplemental content associated with aplurality of objects; and control circuitry configured to: receive asegment of primary content to be displayed on a user equipment devicefor consumption; analyze the received segment to identify an objectwithin the received segment of the primary content; determine whethersupplemental content associated with the identified object is available;in response to determining that supplemental content associated with theidentified object is available: modify the received segment of theprimary content to highlight the identified object within the receivedsegment of the primary content, and display the modified segment of theprimary content on the user equipment device for consumption.
 12. Thesystem of claim 11, wherein the control circuitry, when modifying thereceived segment of the primary content, provides a user selectablelink; and the control circuitry is further configured to: receive dataindicative of a selection of the user selectable link; and in responseto receiving data indicative of the selection of the user selectablelink, display the supplemental content associated with the identifiedobject on the user equipment device for consumption.
 13. The system ofclaim 12, wherein the control circuitry is further configured to: inresponse to receiving data indicative of the selection of the userselectable link: prevent an audio component associated with the primarycontent from being output; and simultaneously display closed-captioningdata associated with the primary content and the supplemental content onthe user equipment device.
 14. The system of claim 11, wherein thecontrol circuitry, when modifying the received segment of the primarycontent to highlight the identified object within the received segmentof the primary content, is configured to: determine whether the receivedsegment of the primary content includes an audio component; and modifythe received segment of the primary content to highlight the identifiedobject in response to determining that the received segment of theprimary content does not include an audio component.
 15. The system ofclaim 12, wherein the supplemental content is a video media asset; andthe control circuitry is further configured to: simultaneously displaythe primary content in a primary window and the supplemental content ona secondary window as an overlay over the primary window.
 16. The methodof claim 12, wherein the control circuitry is further configured to: inresponse to receiving data indicative of the selection of the userselectable link, present a plurality of user selectable options forconsuming the supplemental content.
 17. The system of claim 16, whereinthe control circuitry, when presenting the plurality of user selectableoptions, is configured to present one or more of: a first option tointerrupt playback of the primary content to display the supplementalcontent; a second option to simultaneously display the primary contentand the supplemental content, wherein the supplemental content isdisplayed in a picture-in-picture mode; a third option to transmit thesupplemental content for display on a second user equipment device; anda fourth option to view a purchase offer related to the identifiedobject.
 18. The system of claim 17, wherein consumption of the primarycontent is scheduled to be interrupted to display promotional content;and the control circuitry is further configured to: determine which ofthe user selectable options was selected on the first equipment device;and adjust a frequency of the scheduled promotional content duringconsumption of the primary content based on selection of the userselectable options.
 19. The system of claim 17, wherein consumption ofthe primary content is scheduled to be interrupted to displaypromotional content; and the control circuitry is further configured to:determine which of the user selectable options was selected on the firstequipment device; and adjust a frequency of the scheduled promotionalcontent during consumption of the primary content based on a duration ofinteraction with the supplemental content.
 20. The system of claim 12,wherein the supplemental content to be displayed on the user equipmentdevice is determined based on a content consumption profile associatedwith a first user consuming the primary content. 21-30. (canceled)